ChatGPT_Survey Overview

1 About Survey Details

The purpose of the survey is to investigate students’ experiences and perceptions of generative AI (GEN-AI SPEX Survey!). This survey aims to understand how students like you utilize and perceive Generative AI technologies, including but not limited to tools like ChatGPT.

There are 3 main parts (Familiarity - Benefits and Challenges - Future sections) of the survey including both 5-scale likert type of questions and open ended ones at the end of each section.

Some key aspects;

• University online survey system (Jisc) is used for the survey delivery and management.

• Students who are taking a course in the School of Mathematics are invited to participate in this study.

• The survey was live during the second half of SEM2 (after flexible learning week) until the end of week 10.

• Total number of responses 295 (around 10% of the sample) in 53 running days!

1.1 Data Description

Considered abbreviations listed as below for the degree programme for UG:

• UTMATHM: Mathematics (MA Hons) (UTMATHM)

• UTMMATHEMA1F: Mathematics (MMath Hons) (UTMMATHEMA1F)

• Others: Other, please specify

• UTAPMAT: Applied Mathematics (BSc Hons) (UTAPMAT)

• UTMATST: Mathematics and Statistics (BSc Hons) (UTMATST)

• UTMMAAPPMA1F: Applied Mathematics (MMath) (UTMMAAPPMA1F)

• UTMATMU: Mathematics and Music (BSc Hons) (UTMATMU)

• UTBSCMATBU1F: Mathematics and Business (BSc Hons) (UTBSCMATBU1F)

• UTMATPH: Mathematics and Physics (BSc Hons) (UTMATPH)

• UTBSCMATBI1F: Mathematics and Biology (BSc Hons) (UTBSCMATBI1F)

Considered abbreviations listed as below for the degree programme for PGT:

• PTMSCOPRDS1F: Operational Research with Data Science (MSc) - 1 Year (PTMSCOPRDS1F)

• PTMSCORSCH1F: Operational Research (MSc) (Full-time) (PTMSCORSCH1F)

• PTMSCCOMAM1F: Computational Applied Mathematics (MSc) - 1 Year (Full-time) (PTMSCCOMAM1F)

• PTMSCSTADS1F: Statistics with Data Science (MSc) (PTMSCSTADS1F)

• PTMSCCOMAF1F: Computational Mathematical Finance (MSc) (Full-time) (PTMSCCOMAF1F)

• Others: Other, please specify

In terms of Pre-honours and Honours separation, based on year levels (1-2: Pre Honours, 3-4-5: Honours)

What about the details of Others degree programme? Note that this part needs further data pre-processing

GenAI_Familiarity across different years

GenAI_Familiarity across different degree levels

Based on pre-Hon and Hon separation

1.2 Survey Key Insight

• Almost all of our students report that they have used genAI to some extent.

• More than half of our students have used genAI for:

  • asking technical questions ();
  • asking quick questions when stuck on a problem ();
  • explaining concepts (including generating toy examples) ();
  • asking general knowledge questions or advice ().

• Less than half of our students report that they have used genAI for:

  • checking solutions ();
  • solving homework ();
  • writing essays ().

• Some students mentioned explicitly that they had used genAI tools for

  • coding,
  • summarising,
  • proofreading,
  • producing artwork, or
  • simply “messing around with them for fun”.

2 Familiarity

• Item 1: I believe advancements in generative AI tools will significantly enhance my personal and academic growth.

Item 2: I prefer traditional search engines over Generative AI tools for information.

• Item 3: I am confident in my ability to use Generative AI tools effectively.

• Item 4: Generative AI tools are essential in my self-learning process.

• Item 5: I am not concerned with staying updated on Generative AI advancements.

• Item 6: Responses from Generative AI tools often lack reliability and contextual appropriateness.

• Item 7: Generative AI tools are accessible and user-friendly, regardless of one’s economic or technical background.

• Item 8: I lack understanding of data security and privacy concerns in using Generative AI tools.

3 Benefits and Challenges

Considered items are given below;

• Item 1: I believe Generative AI tools can help students save time.

• Item 2: I don’t think Generative AI tools can help me write more efficiently.

• Item 3: I think Generative AI tools can provide me with personalized and immediate feedback.

• Item 4: Generative AI tools are as effective as human instructors in guiding coursework.

• Item 5: Using Generative AI tools boosts my development of transferable skills.

• Item 6: My use of Generative AI tools does not significantly involve critical thinking.

• Item 7: Generative AI tools provide little assistance in numerical and quantitative subjects.

• Item 8: Generative AI tools effectively overcome language barriers in education.

• Item 9: Using Generative AI for homework does not diminish the educational value.

4 Future

Considered items given below;

• Item 1: The possible integration of Generative AI tools in higher education will have a negative impact on teaching and learning.

• Item 2: The use of Generative AI tools in academic settings should be allowed.

• Item 3: Generative AI tools could potentially substitute for some functions of human teachers and tutors.

• Item 4: It is unlikely that personal Generative AI assistants will become widespread.

• Item 5: Proficiency in Generative AI tools will be crucial for competitiveness in future job markets.

• Item 6: AI will dramatically influence work, society, and daily life in the next 5 years.

4.0.1 Some comments from students!

• Why have you chosen not to use Generative AI tools?

## "teach me"

## "As a maths student, chat gpt is quite unreliable and gives me incorrect solutions"

• Do you have any additional comments or opinions about familiarity with Generative AI tools?

## "Advanced models, like GPT-4, are incredibly useful for explaining difficult concepts like theorems"

## "I think it can be a great tool, but it must be used with understanding. For example, it is good for explaining steps on how to solve something, but not solving it itself, as it makes errors."

## "Generative AI is a useless piece of rubbish which is polluting the internet with incorrect and valueless dreck. Actually useful information is becoming increasingly hard to find, with it almost being a necessity to filter out all content written in the last few years."

• Do you have any additional comments or opinions about the benefits and challenges of Generative AI tools?

## "GenAI should be used as a tool and "study-buddy" to assist students in making extraordinary work, not for simply producing mediocre work."

## "I think that AI can be beneficial in the education process as long as you are using it as a tool to help you do work rather than it doing the work for you."

## "The challenge of allowing said tools in an academic context is regulating it such that students do not misuse it. That is the biggest issue with it. It is often hard to tell when something is written by an AI. Also in numerical and quantitative subjects, AI is frankly quite bad at giving correct answers and that is a reason why AI issues are less prevalent in STEM compared to humanities."

• Do you have any additional future predictions about Generative AI tools?

## "I would like to regard using Generative AI for study and work as a trend rather than cheating behaviour as future market will definitely has high demand of AI agency."

## "In less than 3 years, gen AI language models will be able to complete any piece of written coursework (including maths problems), in any subject, to a first-class grade standard."

## "I believe that AI will become a personal assistant for almost every person"